Add support for BEGINFILE and ENDFILE rules#515
Conversation
Implement gawk's BEGINFILE/ENDFILE special patterns, the nextfile statement, and the ERRNO and ARGIND special variables: - Parser: BEGINFILE/ENDFILE/nextfile keywords, per-file input-loop tuple layout (NEXT_FILE / CONSUME_FILE_INPUT scaffolding emitted only when these features are used, so existing programs keep the flat layout), and gawk-compatible compile-time restrictions (next rejected in BEGINFILE/ENDFILE, nextfile rejected in BEGIN/END/ENDFILE). - Runtime: per-file stepping in StreamInputSource with FILENAME, FNR=0, cleared $0, ARGIND, and ERRNO on open failure; open failures become non-fatal only when a BEGINFILE rule skips the file with nextfile; nextfile unwinds user-defined function calls like exit does; non-redirected getline is fatal inside BEGINFILE/ENDFILE. - New opcodes appended at the end of the Opcode enum to preserve precompiled tuple (-L) compatibility. - BEGINFILE/ENDFILE are not special in POSIX mode (--posix), as in gawk. - Lexer: a slash inside a regexp bracket expression (/[/]/) no longer terminates the literal, which beginfile1.awk requires; this also makes the vendored gawk regexpbrack test pass. - FNR now counts records read from standard input (POSIX: FNR is the record number in the current input file); custom InputSource implementations keep the documented isFromFilenameList() contract. - Tests: BeginFileEndFileTest (25 cases), re-enabled the POSIX nextfile conformance test, new stdin-FNR conformance tests, and wired the vendored beginfile1 fixture plus six beginfile2 sub-tests into GawkExtensionIT (beginfile1 matches beginfile1.ok byte for byte). - Docs: README, cli.md, cli-reference.md, extensions.md, java.md, java-input.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 697ebee809
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
gawk resets ERRNO to the empty string whenever the next main input (file operand or stdin) is selected successfully, not only on the BEGINFILE per-file path. Mirror that in the ordinary input flow and pin the behavior with unit tests. Addresses review feedback on #515. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@codex Please review again |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4622583816
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…E/ENDFILE rules exist A non-redirected getline (or getline var) in an ordinary action could open the next ARGV file directly through the auto-advancing input path, skipping the ENDFILE rules of the old file and the BEGINFILE rules of the new one, and later running ENDFILE with the wrong FILENAME. While the per-file main input loop is active, only the loop itself may now cross file boundaries: getline in an action reports end-of-input at the end of the current file, and the loop then fires ENDFILE/BEGINFILE in order before processing the next file's records. In BEGIN and END rules getline keeps streaming across the input, and programs without BEGINFILE/ENDFILE rules (including nextfile-only programs) keep the classic AWK file-crossing behavior. Addresses review feedback on #515. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@codex Please review again. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df930a4194
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…p brackets - A name=value operand between two input files no longer increments NR: per POSIX (and gawk), NR only counts records actually read. The bump existed in both the streaming and the per-file input paths and made NR off by one per assignment operand; the unit test that pinned the old behavior now asserts the gawk-compatible count. - The regexp lexer keeps bracket mode through POSIX character class, collating element, and equivalence subexpressions ([:digit:], [.x.], [=e=]), so a slash after such a class (e.g. /[[:digit:]/]/) no longer terminates the regexp literal early. Addresses review feedback on #515. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@codex please review once more. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 48e6017f74
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Like gawk --posix: ERRNO and ARGIND are gawk extensions, so POSIX mode compiles them as plain global variables — usable as function parameters, seeded normally by -v and name=value assignments, and untouched by the input machinery at file boundaries. Outside POSIX mode they remain JRT-managed special variables. Addresses review feedback on #515. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Introduce a "Support for POSIX AWK and Gawk" section listing the gawk-specific features in one place, and drop the redundant skipped test_beginfile2 umbrella case in favor of a comment on the transcribed test_beginfile2_test* group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Share the gawk-only special-variable predicate (ERRNO/ARGIND) between AwkParser and AVM through JRT.isGawkOnlySpecialVariable, so the two posix gates cannot drift apart. - Reject direct non-redirected getline inside BEGINFILE/ENDFILE at compile time; the runtime check remains for uses reached through user-defined functions, where the calling rule is unknowable statically (documented on checkGetlineAllowed and executeNextfile, which follow the same compile-time/runtime split as gawk). - Drop the redundant name=value application from CONSUME_FILE_INPUT (NEXT_FILE always runs first) and document that the hook exists only for custom InputSources, which have no ARGV traversal. - Deduplicate the ARGV entry enumeration shared by populateArgv and the synthetic getARGV accessor, preserving populateArgv's Integer keys on purpose: host-injected plain ARGV maps use Long keys and must keep precedence during traversal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
name=value runtime arguments belong to the ARGV file-list traversal, which a custom InputSource replaces entirely: stop applying them before the first record. They remain visible as plain ARGV entries; -v-style variables or putVariable(...) are the supported ways to pass values. Removes the applyInputSourceFilelistAssignmentsIfNeeded hook, its one-shot flag, and the setFilelistVariable/NameValueAssignment helper chain; the two tests that pinned the old behavior now pin the new contract, which is also documented in java-input.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hten the loop Address review feedback on the BEGINFILE/ENDFILE internals: - Drop the SET_ENDFILE_ADDRESS / SET_NEXTFILE_ADDRESS opcodes: the ENDFILE and NEXT_FILE addresses are now properties of AwkTuples, read once by the AVM when it installs the program. Because no tuple may reference them anymore, they are remapped explicitly by the optimizer and seeded as reachability roots so dead-code elimination never removes the sections a runtime nextfile can jump to. - Tighten the per-file loop: the begin_file label now sits on the single NEXT_FILE tuple and the ENDFILE section loops back with one GOTO, removing the second NEXT_FILE tuple (and the two SET tuples) from every per-file program. - Clarify the POSIX gating of gawk-only specials: the parser's isSpecialVariableName and the AVM's isManagedSpecialVariable now express "special in the current mode" positively, replacing the confusing isPosixOrdinary* helpers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the SET_EXIT_ADDRESS opcode: like the ENDFILE/NEXT_FILE addresses, the END-blocks address is now a property of AwkTuples, read once by the AVM when it installs the program, remapped explicitly by the optimizer, and seeded as a reachability root so the END section survives dead-code elimination even when only a runtime exit can reach it. The intermediate serialVersionUID is bumped to 3; precompiled tuple files remain tied to the Jawk version that produced them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the deliberate Integer-key trick in populateArgv: ARGV entries now always use Long keys, the canonical AWK array key form, on both the materialization and the synthetic-accessor paths. The precedence of a host-supplied ARGV over the operand list is now explicit — populateArgv leaves a non-empty injected map untouched — instead of depending on the key-type mismatch, which only protected plain maps with Long keys and silently clobbered Integer-keyed or AssocArray-backed injections. A new test pins the now key-type-independent behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes #457.
What changed
Jawk now parses and executes gawk's
BEGINFILE/ENDFILEspecial patterns, together with thenextfilestatement and theERRNO/ARGINDspecial variables.AwkParser): newBEGINFILE/ENDFILE/nextfilekeywords and rule ASTs. When any of these features is used, the main input loop is compiled with the per-file tuple layout from the issue (NEXT_FILEbeforebegin_file,CONSUME_FILE_INPUT → end_fileat each file's EOF, ENDFILE rules, thenNEXT_FILE → no_more_inputfalling into the END blocks). Programs that don't use these features keep the exact previous flat layout, so there is zero overhead for existing scripts.AVM,JRT,StreamInputSource): per-file stepping with gawk-compatibleFILENAME,FNR = 0, cleared$0,ARGIND, andERRNOon open failure (No such file or directory/Is a directory, synthesized deterministically cross-platform). A file that cannot be opened is non-fatal only if a BEGINFILE rule skips it withnextfile; otherwise the usual gawk-style fatal error fires.nextfileworks in ordinary rules and BEGINFILE — including from user-defined functions (it unwinds the runtime/operand stacks the same wayexitdoes) — and runs ENDFILE for successfully opened files while skipping it for failed ones, matching the vendored.okfixtures.exitinside the hooks jumps to the END blocks.nextis rejected inside BEGINFILE/ENDFILE andnextfileinside BEGIN/END/ENDFILE (compile time when direct, runtime when reached through a function); non-redirectedgetlineis a fatal error inside both hooks, redirected forms are allowed.NEXT_FILE,CONSUME_FILE_INPUT,EXEC_NEXTFILE,SET_ENDFILE_ADDRESS,SET_NEXTFILE_ADDRESS,PUSH/ASSIGN_ERRNO,PUSH/ASSIGN_ARGIND— appended at the end of theOpcodeenum becausefromIdresolves by ordinal, preserving precompiled tuple (-L) compatibility.BEGINFILEandENDFILEare not special under--posix/setPosix(true)and lex as ordinary identifiers.nextfilestays available in both modes (POSIX.1-2024 standardizes it)./inside a regexp bracket expression (/.*[/]/, used bybeginfile1.awk) no longer terminates the literal. This also flipped the vendored gawkregexpbrackIT from error to pass.printf 'a\nb\n' | jawk '{print FNR}'now prints1 2instead of0 0. CustomInputSourceimplementations keep the documentedisFromFilenameList()contract.Tests
BeginFileEndFileTest(25 cases): execution order across files, empty files, stdin, ERRNO +nextfileskip, fatal open failures,nextfilefrom functions and without hooks, all restriction diagnostics, ARGIND,$0clearing,exitin hooks, POSIX-mode non-specialness, bracket-slash regexp.nextfileconformance test; added stdin-FNR conformance tests and anInputSourceTestguard for the custom-source FNR contract.GawkExtensionIT#test_beginfile1now runs the vendored fixture for real and matchesbeginfile1.okbyte for byte (including the ERRNO lines for.and a missing file). Sixbeginfile2sub-tests (2, 5, 9a, 9b, 12, 15) extract their programs and expected transcripts from the vendoredbeginfile2.in/.okand pass. The remaining sub-tests rely on non-redirectedgetlinein BEGIN/END driving the BEGINFILE/ENDFILE hooks mid-statement (or compare non-zero gawk CLI transcripts) — that reentrancy isn't reproducible in the tuple VM; the umbrella skip documents it.getfile/readfile2need gawk's dynamic extension API (@load,get_file()), which Jawk doesn't provide; their skip reasons now say so.Verification
mvn verify site: BUILD SUCCESS; checkstyle / PMD / SpotBugs at 0 findings; 543 unit tests pass.regexpbrackturning green; zero regressions across the 939 pre-existing cases, including after the FNR change.Notes for reviewers
test_beginfile2skip): non-redirectedgetlinein BEGIN/END reads ahead in the main input without firing BEGINFILE/ENDFILE; the loop then adopts the already-open file instead of reopening it.$0and the fields are cleared in BEGINFILE.🤖 Generated with Claude Code